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' can be derived. Indices' of consiotBiiC^ ^re developed which compare the 

n^ber of dominance relations in a matrix to tfJe number that would occur 
' - if it were perfectly consistent and- to the nusnber In a random matrix. . 
• The indices of consistency are shown to relate to test yariaw:e, the 

Kuder-Richardson.formulas, and Loevinger's homogeneity indices. Indices 

of completeness are also proposed. 
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A Basic Test Theory Generalizable to Tailored Testing , 

V " ' . •• • ' . 

t ■ 

The available methods ov strategies for tailored testing seem not to 
Include provisions for a self-eontained evaluatidn of the test. They seem 
to make little provision for estimating anything atialagous to a Kud««;i^i- 
chardson (Gulliksen, 1950) or alpha (Cronbach, 1950) reliability of the in- 
strument itself in the ta-ilored context. Rather, they assume the availa- 
bility of precise item statistictS derived f rom ot}ier sources, , The basic 
obstacle to deriving such consistency ini^brmation comes from the difr'iculty 
* of handling the incomplete nature' of the item response data'^lfi the tailored 
or computer interactive testing case. An atteapt>,will be made here to pro- 



vide ways of filling this need by suggesting 'a^ way to conceptualize the', 
system. ' 

There is a point of vi^w which lo^s on testing as an or<^/eting process, 
ordering persons with respect to each pther or ordering persons , relative to 
items or tasks. The fact that what Coombs (1964) calls "a dominance rela-- 
tion between members of different sets" can lead to a formally consistent 
ordering has |)een known for a^cbnsiderable time. Guttman (1941, 1950) was 
the major pioneer, but the work of Loevinger (1947, 1948) repeated and ex- 
pondtd on many of those principles. 

Recently, there has been a revived interest In ordinal testing concepts, 
e.g. , -Ducamp* and Falmagne (1969) Airasian and Bart (1973), ^Bart and Kruo 
(197a), Krus (1974), Cliff (1975). These papers include ref ormulatiorfs and 
elaborations on the use of binary (right-wrong) information to develop or- 
ders ^ and they lead to the formulaMons presented here. It can be phown that 
a Guttman scale leads to a Joiivir^artial-order of persons and items. Exact 
'Guttman scales — like other partial orders— rarely exist in nature, and then 



^ only in artificial ehvircjnments, li^t. approximations to them are not uncom- 
^mon» The 'ddegree to which a l^en matrix of relations resembles a partial 
order can be measured, and these measures will be proposed as^ indices of 
the eonsiatoncx of measurement of a test. In the case of complete data, 
B0tm of fhesis measures will be found ^to be related to the familiar Kuder- 
Richardson (KR) formulas (Gulliksen, 1950) and to some proposed by Loevin- 
^ger (1947) • ♦ New interpretations of those formulas result. Some of the 
measureo will be new, and it will be contended that they are more relevant, 
than the* traditional ones, as measures pf order, ^ , ^ 

The formulations lead quite directly to gei^eralizations^whi^h are ap- 
plicable to the incomplete data — tailored or computer-interactive-"-ca0e. 
That ioi measures of the internal consistency of responses will'ba devel-^ 
oped which are entirely self-contained in the tailored 'testing prpceos ra- 
ther than relying, on outside information. 

Notation Basic P;rinciples y 
The basis -of recent developments in tests as ordering instruments ib , 

^ t^^^noideration of the score matrix as ^items-plhs-persons by items-plus- 
persotio rather than s^.mply items b^y persops. Then a l^l^ndioates a' domi- 
nance relation -of row element over column element, regardless of whether 
item or person is Mominant. The following conventions are used in later 
developments * . 

TliG matrix S contains item-person responses with s^^ » 1 if person' i 



answers 



item j correctly; i <^ 1, 2, ...n; j" « 2,, ...x'. 'S' is a ma^:rix 



of item-person responses with ° 1 if person i answers item j,. incor- 
rectly. * * 



6 



A is a supermatrix: 



. (1) 



0 'g'' 
S 0 



where 0 Indicate^s null sections apd-'§^' i'a the transpose of Thus the el^., 
ements of A indicate dominance relations of row element over column alement , ^ 

< ■ 

regardless of whether the elements^are items or persons. Item-item and per^^ 
son-peraon relations are .always 0-0 because there is no direct observation ^ 

1! 

of these relations. *. 

For cpmplete data^, S and are' complements, ' 

■ (2) ' 8,, - 1 - 



ij 

In matrices 



ij 



, (3) S - 1 4' - S, 

* . mi l* . . - 

I . ■ ' ' ' \ . ' 

where Jj, as elsewhere J>61ow, derates a column of imitieo* The compllGmentary 
relation does not hold^in t^illored testing where some elctnents are zero in i 
both matrices because tiie person was not presented the item. ^ 
The ordinary matrix product of A wi~th itself is denoted * 



J4) a2 i« ! N 0 
' 0 X 



Note that all even powers of A have the generrii form of (4) whereas the 
odd powers resemble (1)^ Th^ scalar notation a. ^denotes the'ij element of . 
A^. Jn (4), where , . * ' 



(5) 



the element n^^^ is equal to "the number of persons who get *k wrong and j righ^,. 
I.e., the nuiober of times itbm k dominates j. Similarly 



\ 



* , (6) X « S ) • . • \^ 

*is a petson-dominance matrix, with element x., , and contains the number of 

- ^ in 

times person 1 dominates person h, i»e» A'the i^iimber that i gets right and 
J h wrong » A ' • / 

Total scores are also used: ' 

(7) ' X, * 51 and. • 
1 

A/. 



, (8) % - SI ■ . = , - ♦ • 

V • f ■ . 

Similarly, ' *. ^ . 

(9) Nj ° VS and ^ • 

o \ ' ♦ ^ . 

(10)^ S^f - . / • 

Th^ vector should be noted as a difficulty scor<^, the number of, i- 

tern "wins," not the' number passing" as is more typically used. "^3^^. 
For complete data, ^ 



and 



(11) x^ - X - If^, 

(12) iij - n - 'iij, ■ " - ^ 

but again thq 'relations do not hold for the misoiixg data ease. 

^ Use io made of the "BoolCiin powers" of A as well as the numerical onea^ 
particularly in considering the logical properties of the relationa. Here, 
the matrix multiplication uses Boplean arithmetic in which 1-+ 1 » 1 + 
0 + 1 « 1, 0 + ,0 « .0; 1 • 1 * 1, 0 ' 1 « 1 • 0 ^ 0, 0 • 0 ^ 0. Such powers 
are (lenoted by having the exponent in parentheses 



(13) a(^> 



0 
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where D and Y are fihe logical counterparts of N and X, respectively. 

Still a third form of pdWers of A is used, A^* in which binary en- 
tries record the relation between the Elements <^f and some threshold, 
values for them.' Let be an n -t- x by n + x matrix of threshold or cri- 
terion values for A^. Then ^ A , ' 

(14) ,aP* « (aP > ZP). ' ° 

That is, a*^^p -^^^ 1 if a^^^ > and zero otherwise^ On oocastpn we may 

have cause to refer t6. the aubmatrices of A^* as S*, N*, or X*^ iind tb^ 
• their elements s*.v^ etc. \« ' . ' * \ . ' 

Thus A^(P*) « A^* if zP o 0. ^he matrix Z^ could- also be a constant 

such , as 2.0 If , for exampiei it were felt that a,t least two connecting rp- 

■ * . • . ' > ' •. 

lations * should >e necessary to imply that i dominates h . Ktus (1974) in ( 

effect defines /^^j^p as the number ofi -relations necessary to shbw that i 

"significantly" dominates h. . • 

The foregoing are the main notational conventions that are 'used be- 
\ 

lox7. A few additional ones are introduced as needed. ' . 

, Indices of Orderliness . * 

Person i "dominates" person h when he gets an itdm tight which h gets 

\ ^ . 

V " . ' • • 

wrong ."^ Item j dominates item k'when there is a person who gets j wrong' 

■ . . 

and k right. .The dominance relations are consistent insofar ab they occur 
in only one direction, i,e., are asymmetric rather than there being' in- 
dtanceo of both j dominating k and the reverse. This is what is meant by 
reproducibilitv ta a* Guttmati scale "(Guttmani 1950). If the* objective of 
;the measuremGnt process is to order persons or items or both, this ordering 
should be complete as well as consistent, at least in most Instances, for 



a highly Incomplete ordering la almost as useless as a highly Inconais- 
^tent one* Consequently ; indices of completeness as well as cotiisistency 
'are coXied for^ . 

A useful consequence^of th(i*^*use of the complete matrix A rajther than 
simply the rightg. matrix S is that every index derived for items implies 
'a parallel one for persons. Tlxis is a complete duality as substitution 
of Items for persons and°vice versa in any ^fprmula. or theorem will result 
in a, legitimate parallel formula or theorem. Indices derived, for items 
will often turn out, to be related to traditional 'statistics! and formulas ^ 

fl , • - » r 

although it will often be argued that the ;>parallel ones for persons ard 
actually more relevant to the purposes of tesjt^ng* 
The number of dominance relations 

" For the most part we will concentrate on equations relating to the ny 



.merical mode of deration implied by Equations 4, 5, and 6- although the . 
more ^qualitative formulations of Equations 13 and 14 will be returned to 
later so that some analagous measures can be proposed, ^ . 

The primary dominance information is simply the number of elemdnts in 
the peraon matrix N .and the item matrix X. We denote these quantitieo v 
and 8, 'respectively: 



(15) ' V » ?En , * and 

(16) ' g - SEx^jjV. 



In matriees , those are 

(17) . V ° l'(^'S)l , and 
; (18) 8 » i'(S^')l > 



Id 



4 « 



-7- 



. reap&ctiviely, so that relocation of the parentheses' matces .it clear , that 



(15) 
(ZO) 



Minimum 



V » J^'X ' and 
i 



er of dominance relatlonfl 



^ Now Suppose the sum takes place only above the diagonals of the ma- 
tricea: , V 



m 



S S n 
J k>j 



Jk * 



(22) 



i h>i 




If the items or persons are' in the appropriate order, and the data are per*^ 
^fectly consistent, then the matrices" will he upper triangular, ani v will 
equal and g.will equal g^, otherwise they will be larget^. This auggeoto 
tlie ratios g^/g and v^/v as consial:eficy lndic{5a> but they have the dr/kwbaclt 
that they will tend to be fairly close tp tmlty for moot data, as the coef- 
ficient reproducibility is. They,4d, however, have^ the advantages over ihe 
latter of having tetmo with a clear interpretation and being straiohtf or- ' 
ward to compute, . ^ ' ^ 

The indices v^' and g^ can be computed from tha marfiinalo without ref- 
erence to^the n^j^ at all. . If the data are consistent, n^^ ^ 0 for h >. j. 
Subtracting from each tqrm of (21) gives 



(23) 

But n^j^ - nj^j^. nj - nj^, so 



(24). V » E E (n - n^) 



■ ■ 

This reduc^- to 



(23) 



or 



(26) 
Similarly, 
(27) 




v^> (x+ l)Enj;- ^ 




'^Thua , If .Is hot 'necessary to -abtually have the N or 1^ inatrieeo to com- 



•pute y,.v^, g, or g^. 



An obvlouo eorreetioft torn to apply to g"^/g lo to aoouiEe that thero 
13 no'^prdor, everything la equal. Then n,, and^, ore botH eoti.BiatGo of 
^the oame quantity, vjk, "and x^^ and. Xj^^ oimllarly eotimate x^^. Tiio cot-' 
laatSj^rc combined. by averagdirig, the quantefty of Intereat would tiien be ^ 



fchq de8rG"e\^p, which ^)e^n^j^ a&d . x^^^ exceed theoe eotlmateo, replacing 



- quafiioi^a .ZJt |»d with .^ . . , 



(29) 8^»" - 2 S 
. 1 h>l 



^Ih - -^^ih \l> 



When thc'otpo are diotrlbutcd, this pcGulto in 
. (30) ' v^, - v^ - ^^ ; • . . * 

, . ^- (31) 8^, - •> im . 

The maxiiaa for theoG two qisantltico would bo% and kg, roapGetlvely. The 
iwo of ehooo latfior quantities ao dolioalaatoro leado to the fellowiag, Indlceb 



r 



of consistency: 



032) ' < '^c^ -^^m - 

■ ■■. ■ ■■■.«v.. . - 



^Is an Itidex/of ^consistency of Item ordering for* a teati^and v 

. ■ (33) c -^8m^ 1 . • ' ' \ ■ • ' ' . ' 

. P "7"' . ^) ■ ■ ■ 

; ■ • ■ ■ ~ t> ' - • ' » - . ■ . ' . • - : • • 

... , ' . ■ • , ** ' - - 

is an Index 'of tjonslstency of pegrson ordering for a test , 
AppllcatiLon to Incomplete Data ^ s 

■ ■ , ^lir^. :, - --r -^.r''^ ■■ ■ • 

Ml*of /Bi^ calculations ned;e8sary for c , and c dan be carried out 
for incomplete as well as complete data, but the definitional equations 
I5V 1^, 21, dnd 22 fflU3t be usecf instead of the simplifidations, .The lat-^ 
ter, as rioted lifter, can be shown to have relationships -to- more tradi- 
tional psychotoe'tric'' indices and also to the H measures, of Loevingfeir 

■ ■ ! ' ■ . -Jr ' ■ < ■ 

(1947, 1948) • 



There ni^y be "some concern' over how the order is def in^iil^o'^' t the 
above-diagonal sutn^ are as large as possible. Foi^ eoi^ipiete (lata, order- 
ing j^o that i <'h implies > x^^ and j < k implies ny » nj^ will lead to 
' largest values for g and v* than any alternative. To prove this, let 
\t be the same as (21) except that items j and* k are interchanged in 
>^ the order. This means that cert.aiti elements which were above the diag- 
onal, have been -replaced by their symmetric counterparts: Assuming, j < k, 

k-1 k-1 k-1 k-1 

<3*> \t " \ J Aa : , ^"ak - ,^"jb V 

'a=j , b«j -^^ b=j ^ 



Then 

k-1 k-1 

(35) 'V„: = V 

a=j b-j 



But for complete^data 



do 



k-1 • i ' k-1 



<37). , v^^ = v 



^+ (k - j)nj^ - . Z ^ . j)n > t i^ y 



The summed duantiti'es are identical, so v - v will *be positive if 

iL n. ±B, Thus the maximum g and v are attained when items are in 
K J m m V 

order of difficulty and persons are in order of score,* respecfeL/ely. 

The- procedure as not as clear-^cut wh^h the da£a are incomplete, but* 
the fdrmulatron used in the proof c^n bd» used to define a maximizing or- 
der« Gi^en the items in an^ ^rUitrary order, begin by calculating for 

that order. For j « 1 and k = x calculate v - v\^' using the indicated 

■ * 

sums, if the difference is negative permute j and k. Then repedt the 
process for all combinations of j and k for which k - j, f x - 2, permut- 
ing those items with negative differences. Then repeat with k - j x - 3 
and s^o on. The "outside-^in" character of the process would appear to guar 



aiitee that the final value for v^ is the largest possible one, but no 
proof of this conclusion is offered here. 

The V index is calculated pn the order resulting from the above 
•I 'm 

procedure and is recommende4 as an overall index of item consistency. 

^ ■ ■ . ■ ■ ' ■ . . ■ 

The g index for person consistency would be calculated' in a'parallel 
way and used to measure the even moi^e important property of consistency 
in ordering of persons. ; 



' Completeness of Ordering 

■• . " ■ W . . ■ . „ 

Basic concepts • - ' - 

y\ The igompleteness of the ordering process is only slightly less impor-^ 
tant than its consistency, particuljarly as far as persons are ^^^ticariaed. 
After all, if persons are merely divided into two classes, one cb|a|l's'^^lg 
of . a sinjgle individuaL, then the fact that this is done with perfect con- 
sistency is of only minor comfort. Indices of completeness are thus im- 
portant adjuncts to indices of consistency. Such indices are xorigldered 

o' • • . . ' ^ - - ■ . . . ■ . * , '. 

^ • - ' ■ . .. • - ■ ' • ' ■ " 

in this section, ' - ^ 

There are a number of ways in which completeness can he defined, but 
the* most "obyious ihdex is the proportion of pairs for which x^^^ ^* 
A more general index is the proportion of pairs^ for which l^^^j^^ "" ^^^^l ^ ^iji2 

wheje the latter is a criterion or threshold value as desciribed in the ear- 
lier section on notation. That is, we^deflne indices of completeness of 
person and item ordering as r and r^, respectively, where 

^ 55n(n - 1) ; , 

and ^ : 

, _ t . kpcCx -1) . . • : . 

For complete data and criterion values of zero, this is simply the num- 
ber 6,f pairs of scores and difficulties that are not tied. In that case>, 
the t:wo indices must influence each other. If there are y different diffi^ 
Gulty levels and the data are'^perfec^ly consistent, then the largest possi-"". 
ble number of different score levels is y + 1. If y 1 < n, let , 



^^n^let J)e the largest int^ger < b. Th^n the largest posaible value 

* for r^ occurs when some I. individuals are tied at each of x(b - L ) . 
scores and I^^ + 1 are tied at ^adh of the/ x(I + 1 b) remaining ones, 

* This can be used to derive an URper bound for the number i5f diffe^i^t 
p03€iible 8(iores given the item difficultfies. This c^an be i^sed as a de- 
nominator of (i'S) in place of ^$nCn'- I)i A K:<pres ponding adjustment can 
be made in the denominator of (39) tp aidjust v^ for the scopes, • 

: * ^; ' Extended Implied Orders V * * 

The preceding developments and proposals have, been presented as ones 

which would apply to dominance matrices whictx were derived froA either 

* " • - ■ \ * ■ ■ ■ 
complete score matrices or incomplete ones, but in either case irom thosd 

item-person relations which were actually present in the data. Recently 

it has been suggested ((Uiff, I975> that the incomplete scor^ matrices 

employed in interactive testing can be raised to higher powers than.tWo, 

These higher powers can be used to complete ^the dominance matrices N and 

X when the score matrices are so sparse that they do not do so themselveis. 

The process can also b^ used to compute the odd powers of A which will, re-^ 

suit in completion of the score matrices themselves. The indices proposed 

here can be applied to dominance matrices which are derived from 8uch im- 

plied score matrices as well as f rom N and X themselves*. 

There are several different ways one could proceed in Completing dom- 



inance matrices. The simplest is to calculate SN^ and' SX^ for some valtie 

p P 



and' SXP 
P 

of p and then calculate the indices from these matrices* But. in effect 
this is as if the scoro matrices were other than binary because, e.g.. 



' -13- 
:<41) N3 • (^•gS'»)(j5^»S), 
and the two factors on the right ;can be thought of as implied scpre ma*- 
trices, and they need not be binary.. ^ 

A more consi&tent approach is to calculate binary implied score ma- 
^ trices.. By the notation suggested earlier, for ^^arople, A^* is such a 
binary matrix whos 6 non-ze,ro elements contain items which are right or^^ 
wrong by implication ' as well as directly:. Corresponding N' and X matri-^ ^'^ 
ces can then be computed and the indices of consistency and completenfess 
calculated from them. Similarly, A^*, A^*, .V. could be derived using more 
remote implications, and the corresponding N and X matrices used to ^ive 
indices for them as well* thus the. indices would be used for implied 
score matrices as Well as observed 'pnes. . ^ 

K The question 0$^ how the values in the .critferion*'matriX' Z are arrived 
at is not dealt with here. This is the major statistical inference jirob- 
lem in tailored testin^^ how it is decided that ;a response to an item is 
predictable* on the basis of 6ther responses * Different methods of doing 
th^t lead to different implied score matrices and'the different implied 
score matrices will have different degrees of consistency. Until more de- 
tailed analysis shows, otherwise, it muat be borne in mind that some me- 
thods might lead to spurious degrees of apparent consittency in the order- 
ing. One* of the desirable 'chq^rract eristics of a tailored testiilg method 
would be that it not lead to such spQriousness, but the problem of how to 
ensure that ^aneems a difficult one. 
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. Ordinal Information and Its Rep.ation to Traditional Psychlometrics 
Variance and Dominance 

In the^ complete case, the traditional psychometric indices of loca- 
tion, dispersion, and internal cdnsistencyi-taxl be r^ated to dominanqe 
measuresj particularly those related to items. Thia is to be e^cpected in 

View of the fact that then the complement matrix^ can^be derived from S 

# ' " • ' •■ ■ 

/-by numerical meians (Equation 3). lislng equations (11) and (19) * 



(42) 



Then since Ex^ ■ ipc and Ex^ ilS?- + no 
(43) V -..nX^x - x) - naj' f 

and 

(44) 



« S(x - 1^ - 5 
X n 



, 3hows the relation J>iatween variance an4 item dominance- relations . The re- 

^ .' - .. . ' ' 

lation is , complementary rathet ^^han direct , which is counterintuitive at 
f ir6t glance. 

^The maxitniiim possible value for/a^ occurs when the score distribution 

X 

is dichotomous, and then it is T(3rjK x) . In this case v will equal zero; 
there will be no item dopdn'ance relations*. This jpaeans 4II items have ex- 
actly the same score profiles ; ^thfey will be equally . difficult but also 
perfectly consistent. If the items are of equal difficulty but not per- 
fectly consistent then cr^ will npt attain this maximum' value. 

Looked at the other ^way, ri5^(x - x) is simply the maximum value v can 
attain, and it will be denqted 



w 



(45)' 

a. 



"v^ ■ nx(x ~ iSO 



18 



In thB present context x(x - x)* is taken as a fixed quantity, but it 
also represents the appropriateness Of the items/ and persona for eadh other 

• / r . - 

in terms of average difficulty and score leVe^. Dominance relations occur 



insofar as items and persons are appropriat^ t^ eat^h other; if itema are 



too easy or hard, then 3c or x - 5E wilL be zero and no domijLnance will occurt 




BofTring this as well ^ as the preceding resjiit in mi1id> variance is propor- 
tjonalto the difference between the number of^ domin ance relations and the 

' ^ ' • — ~n"" — ' — ^ — • — '■ r- ' — ^ 

maximum possible/ number that can occur given the average score > and dlffl- 
culty levels * * o 4 * ' , i 

It may be noted in passing that by^this ihteypr^tation variance is a 
pure number, a iiniti^ess scalar, since v has^g^rsons as units and the con- 
stant of proportionality for- relating it to variance Is n* 
Person dominance . , ^ 

It Ij^^^^haps curious that total score variance, wfiichi is habitually 



thougnt of as relating to discrimination among persons, is found to relate 
to "item dominance rather than person dominance relations: The latter'^ too, 
are related to a more traditional quantity. Consider (20)^asing p^ « n /n: 

(46) Enj(n - n^) « n^Sp^Cl - p^) 

Thus the nuiiJier' of person dominance delations is proportional* to the aura of 
*the item variances, Krus (1974) and Loevihger .(1^47) note t*his as' the re- 
lation between PjW\- Pj) ^"^^1 the number of discriminations made .by the i- 
tem# * . . " • ' 

T^e Kuder^-Richardbon formulas ^ 

It is of interest tp interpret .the Kudqir-Ricfiardsdn formulas in terms 
of dominance. Suppose two items are independent. Then the expected value 
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Qf n^^ is 



This can be ua'ed to define v^, an expectation for v corresponding to the 
situation of all items being independent. V 




(48) V - S 2 E(n J , . ■ , 
ov for complete data ' ' ' 

." , ' ' : .• ■■ 

(49) . v; - 1 SEnX -r iSnX . ; . ' 

n jk ^ rt ^ . 

Substitution and simplification leads to 

(50) - nX(x --x) - nSpjd - p^) . - , 
Using our previous expression (43) for v i^tenns of varianceNleads to the 
difference -^a - ^ 

, (51) Vj, - V ■» na^ - nSpj(l - p^) • ^ 

In present notation, KR20 is . ■• 
(52) KR20 - X - 2p^(l - P.,) ) 

3C - 1 ^ 
X 

If numerator and denominator are multiplied by n, (45) and (50) lead direct 

- ■ ■ • ■ ■ > . 

ly to 



. (53) KR20 » X "^c ~ ^ 



1 V 
w 



Thus, aside from the correction x/,(x - 1), KR20 is the ratio of the differ" 
ence between the number of dominonfiQ relations and the number expected by ^ 
chance the difference between v and the maximum possible number > The cor 

v - • " ■ . ■ • 

20 



^ rection has the effe"ct of making the ratio Mnity- when v takes on ±tQ min- 
imum possible value. The foregoing provides still another interpretation 
of KR20/to go with the recent one provided by; Kaiser and. Michael* (19^74) ♦ 

A second variety of chance expectation for n.. merits consideration. 
Suppose all items were riot only independent but of equal difficulty. That 
3is, there were no true dominance relations at all, the observed bnes rep- 
resenting simply random" events • Then > - ^ 

^ (54) E'(njj^) - npd - p) . • 

*for all j and k, 

. ' ■ ' ""^''-^ ■ v^i^ y ■ ■ 

'The obvious estimate of p £s tt/n, so under the&e asaumptionQ 

= (55) E'(n j^) -'iTdi - SX/n 

Then we catf define a aooond chance expectation for v: 

(56) - V • « S" E -> , 

c J kytj^^" *" • ' 

which Is obvioa*ly • 



(57) v^' - x(x - l)1!(n - nT/n' . 



Since xn » nJf here 

(58) v„' - ri3c(x - 3E) - ri3e(x - 1t) /x 



or 



m v'.(i-i)v 

e X w 



Tiien 

(60) v^' - V « nal - nx(x - 1!t)/x . 

$ 

Rcealling KR21 in present^ notation leads to 
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Thus KR21 is the ratio of the differen ce between the nutober d6>o.bsetved 

" ' III II - I I II I . _ 1 ^ I ■ « 

dominances and the nuiaber expected under independence and, equal digficulty 

"■■ • . 

to the difference between the nuniber of observed dominances flnd'tha max-' 
imum possible number > 

The fact that the two principal KR formulas /caa be expreased^in terms 
of consistency of item dominance, whereas consistency pf person is preoum- 
ably the goal of testing leads to an intriguing notion. ^This is -that tl>G 
KR formulas should be replaced, oft at least auppleirfented, by ones which 
express consistency of "person dominance. Denoting these 'as KR^20 an^KR^21, 
the duality of items and persons leads immediately to 



and 



(62) 



(63) 



ICR 20 a n 
P 



n - ,1 



KR-^21 . n 



n - 1 



Here, 



(64) 

1ft vhi?h 
(65) 



« 2.. X E(x^j^) 



i h^i 



X . (x - X. ) 



and g'^, and g. are defined in ways parallel to v*, and v . respectively. 

C o w , . / C W 

In terms of conventional psychometric quantities (0^ being the variance of 



the n^) these arc expressed as 



(66) 



KR 20 
P 



n 

.n - 1 



1 - 



£a^(n - n^) 

n^0^ 
n 
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and . ' 



p . n - 1 



n ^ 



It is couc6ival|le that fomulaa of the KR type eootd be generalizad 

to the tailored caqe, but no way "of doing this has' pr^sented^itseif • Tlie / 

difficulty is in defining an analogue of the variance which is strictly 

comparS&le In complete and incomplete cases* . The* irifeompi'eteness of the 

data gives a flexibility which makes it impossible to define* v 1 the maxi- 

mum possible value of v, In other -than an arbitrary manner The concept 

of a total score is equally pllppery with Incomplete data/ The c Indices, 

on the other hand> depend only on there being orders for items or persons • 

' ■ ^ w , ■ ■ *• . ^ 

They are calcuablc on any order, even an 'arbitrary one. Moreover g thg pjo- 

, ■ . 1. ' ' . . 

cedure outlined above for f-inding an optimum order seem quite straightfor- 

ward.v Hence the e indices are recommended as appropriate indices for ao-- 

'sessing intcritem consistency in both complete and incomplete cases. 

Loevinger 'ljr Indices ^ - . ' ^ . 

• Loevlnger (1947) defines a homogeneity^ index H which depends on V , 

« , . X 

the variance of the present tes^j ^homr' variance of a perfectly con- 
sistent test; and the variance of a test composed of independent i« 
terns. The latter are analagous in concept to v^ and v^, respectively. The 
index turns -out to be mathematically related to them. 

The index is defined by Locvinger (1947, p* 31) as 



horn het 
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In present notation, . 



and 



(70) V : V, 



''hoia^ " l^nEjn^ - n2n^ - CSnj)2j 
It is easily seen from the discussion of th? KR formulas that* 
(71) n(V^'-.V^J -v^-. y . 
Furthermore, from (26) and (SO) ^ 

" . . 

- (72) n(V. - V. J - V - V . C 
hom het' cm ^ 



Therefore, Loevlnger'a index of homogeneity is 



(73) V - V 



H 



c m 



this immediately suggests an index of person consistency defined in terms o/ 
the g indices: • ■ ^- .'^ . ' . " . ' 

' (74) g^ -"i ^ ' . 

H - _ V \ • , . . 

I P ~ ^ ' ' 

The resemblance Q3^,P6rmula (73) to the KR20 formula coupled wl£:h the • 
present concern with dominance relations suggests a* further alternative. If 
there are no true dominance relations. among items, then the sample item 
dominances represent purely chance events, as in the analysis of the KR21 
analogue above (Formula 61). Then an altemativ:e to (68) is ■ 

. (75) H^.' = ^ • - 



V ' - V 
c m 
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where v^' defined as in (56). \ • . . ' / ' 

Still a third hdaogeneity index suggests itself, one. in which v Hja 
used' as an origin rather than v .or v^' : - ' * 



^ (76) h/-v^-v. 



w m 



•there would be corresponding homogeneity indices for persons 

. . Q • • . . . 

(77) H. ' = g^v- g . , ; 



and • ^ — ^ ' ' . 

(78) Hp"^ g^'~8 . • 

The chance expectations ''v and g can be generalized to the incomplete 

/■ / - ' - ' ' ■ . ■ 

case, so IL and H could be used -to evaluate the measurement proceca there. 
That is, the definition eitibodied in Equation (48) woulci' still apply, and 



£t could be used to calculate H. for incomplete data. The Individual. E(n.,) 
are straightforward to compute but could be time-consuming because 'they 
would need td be based on only the persons who took both items ♦ Eoaentially 
n is a variable depending on the j,k pair rather than. a constant. In a 
eoi^puteriscd system the additional requirement Kould not pooe an ir*oupcr- 
able difficulty. Some of the simplicity of the e indices would be loot, 
however, by the Introduction of this compli cation of computing not only v 
and v^ but v^ as well. Thus the c indiccis seem still to offer advantages. 

Tli^^ variations H^' and 11^.*', on the other hand; become much too compli- 
cated to deal with* in the incojmplete case because of the difficulty in de- 
fining V, which was noted above in the discussion of the KR formulco. 
w 




^ Item Infidrmatibff - • • •V 

Item Difficulty ^ ^ , / • * ' - - 

The basic information about items which is of concern<*to thavteat de- 
vclopcr is rthe item difficulty and discriminating power. These are tradi- 
tionally measured by proportion passing, which is 1 - p. In present^ no ta-^ . • 

tlon, and Item-test ^locrial or point^biserial correlation^ respectively • 

* \ ' • « 

H(jre w^ deal with the question of what to use for these purposes in the * 

' ■ ^ ^ . * \ J -'^ 

present 'bont'ext, . . 

The *lllff lenities inherent in measuring difficulty as proportion paos- 

Ing are obvious in the tailored case where the object' of the^ tailoring 
• " ' ■■ * . 

Strategy Is to keep the probability. of correct response as nearly constant*, 
ao possible* This fact is the basic motive here although the case of ' com- 
pJLete'^data la considerlid first. We seek indices whtieh not only make inj;u- 
itive sense In the completii case but also in the tailored one.' 

I 

One of the theoretical innovations of -modern test theory is the plac- 
ing of persons- and items on the same scale. This is ttue not on!%;^f the 
trace.Xlne models (tord and Noviek, 1968), but also of the ordinal model ^ 
used here. The 'introductory discussion made note o^thc fact that iz leads 
to a Joint partial order of 'litems and ptsfsons. Analagono to the scale val« 
ue of items on the ability continuum in the tracelipe models, have the ' 
position of the item In the joint oi^der. , 

To specif y©' this, it^As neeess^ary to know not only the item-peroon rcla-^ 

tlons but the person-person and item-item ones ao well. Here we make use 

• ■ * 

of the relational matrix In the oimpleot (|jioG, take A 4* A^*. Then we 



define ao followo 



1 
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(79) 



■■ ' .1=' 

" + A^*) l - 1\(A 4- a2*)J /(x + n) , 



-23- 



where "the Index 1 runs over both Items, and Individuals*. The score Is 

. ' ' •* ' ' •' ' 1 ■ . 

thus the dlfferenpe between an element's JI«lns" over Items and persons 

and Its losses.* 

■ ' . ' • , ■ , <^ . 

Far Incomplete data. It may be necissa^ .to make, xis'e of a "Boolea^ 

^' Jl' • - ■ t ^" ■ ■■ . ■ • 

sum" operator denoted Here, ]J^a. . Is. tkken in Boblean arithmetic. 



en 



(80) 



,f^^ u. » 
^ip 



^ p p - 



/(n + k) 



cfin be used as a generalized index of the positidn^of the item or person 
in the order. Tf p is limited to unity, then u^^ is proportional to a 
right minus wrong score biy a fail minus pass index. • ' 

, There 9[re a couple of additional points that can be noted.' First, 
items and p^rsoiis are treated together rather than separately or differ- 
ently. Second, u^^ provides an alternative way of defining the appropri^ 




order ;to compute v^ or g^. 



ate^ item or person ox 
Discrimination In^ici^s 

The obvious place to start in defining "a "discrimination ifidexV is 

J ■ ^ ... i 

with' components of the v and v^. The total number of ^ dominances involving 



item J is 
(81) 



The number which are In the appropriate direction are 



(82) 



1 k-j-hl.J'^ 



ERIC 



27 



-24- 



I- then V * JjZtt^ and + JjZv^ , This immediately, suggest? that 

^ la an index of coiMslstency'f or Item i . . 
Comple€e Data ' ^ " 

' For a given item, if there is complete data, 
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(85) ■ - S(l - 8^j)x^ + Z8^^(X - x^) . ' - 

This rediucen to the dlfferencie between the sum of the total scores of v 
/^^thpse who got the Item right and the sum for those who got It wrong, plus 

\ , . ^ ■■■ J/ 

The minimum^ number of dominations for j , v . , Is, the sum of the above 
diagonal entries for that Item: 

k-J *^ k-j+1 

. ■ • . * ■ ■ *'-■•" 

This" can- be expressed In terms of 'the marginals by the sape reasoning as 

for y ' 

^ m 

(87) V - S n^- Z n^ + (n - 2.'+ l)n. . 

<^tj i« computable from item, and person totals when the data is. com- 

■ 'plete; ■ ; ' # , ■ ... , ' . • : 

Item indices of the variety can a^l^ko be used. Recalling Equations 
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(4^) and (49), we define 
(88) 



and 80 . 

(89) V - 1 r 1 

Then a homogeneity index for item j can Ibe defined as v 

(90) H\, - y. V. 




Using the second type of cha&ce expectation used in (54), i.e., thit i- 
ten difficulties are all the same, leads to 



(91) 



- 2n - 2 « 



n(n - H) , 



and an H»j may be defined parallel to H . 



-V 



Surprlringly, Loevinger (194?) did not use an H^j like the above 
to measure item consistency. Rather, she made, an interesting departure, 
in effectfmeaWring the change In g, not v, resulting from deleting the 
item. ThlXis^TO interesting approach and has a good deal to be said 
for it conceptually. It may have been impractical given the computa- 
tional facilities -available at that time to use her approach. On the 
other hand, Loeyinger does use an index similar to (90) in measuring the 
consistency between a pair of items. 

In comparing,, c^j to H^^^ , th'e ' advantage may lie with the latter* since 
the former will tend to be larger as items' difficulties deviate from .5. 
For H^j, on the o^her/lii^d, the bias is, if anything, in the opposite 
rectionU 




Parallel indices can be developed for Individuals. While persons 

are normally 'treated as if they did not vary in their consistency, it 

' . • - , - f 

is quite conceivable that in fact they do. Certainly in achievement 
testa the difficulty of the items i^/large,^y a function of the manner, 
in which the Educational system is organized. Individuals who taka an 
achievement test'N structured for a particular educational system but who 
have participated in a different one may well be expected to have low 
consistency Indices. To take; account of the possibility of different 
consistencies for di'fferent persons, we define 

and % 

* ■ 

where the g indices are defined parallel to the corresponding v. 
Multiple Alternatives '' 

In the foregoing, it has been tacitly assumed that item scoring is 
dichotomous: Right or wrong, endorse or 'reject. This limitation is .not 

m * 

•r 

a neces)9ary aspect of the system, and in many contexts it makes sense to 
consider multiple alternative responses separately as items in their own 
right. There are a^ ntuober of posslb Indifferent cases; methods of treat- 
ing a few of the most obvi^s will b^ suggested here. 

First, there is the multiple choice item. One of the benefits that 
may be expected to atitUe from the computerization of testing is the even- 
tual d^ise of this type, but th^ limitations Imposed by habit to cay 
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nothing 'of investment, mean that it will be around for' some time/ The 

obvious thing to do will ^e tp put in each of the aternatiyes as a sep- 
arate item of the score matriae. IQiis raises, the question of the score 
pattern that: should be used since we use If as well as S. A logical thin^ 
to do is to aet s^^ - I and 'STj^ « 0 foy each correct alternative the in- 
dividual chooses and for each incorrect alternative he does not choose. 



On the other hand, s^^, - 0 and s^^ - 1 fo* each correct alternative he 
fails to choose and each incorrect' alternative he dofes choose. ' 

In the traditional four-choice item where a single alternative must 



be selected, a person selecting the correct alternative would have s - 
for all foUr. possibilities. A person selecting an incorrect alternative 
would have s^j - 0 for the correct alternative and for the incorrect bne 
he chose, but s^^ - 1 for the two incorrect altenfitives he did not se- 
lect. In this way,, "difficulty" and discrimination information could be 
derived for ♦each alternative* 
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A second class of multiple alternatives are those .where the choiceg^ 
form a logical scale of,some*kind (e.g., "never," "occasionally," "fre- ' 
quently," "constantly"). In such, cases, if it is logical, to Assume mono-l 
tonicity for the alternatives, a person who chooses an alternative would 
be assumed to. have also chosen all less extreme alternatives. A concep- 
tually similar case is the timed item, which will be the. replacement for 
the speeded test if the computer ever does take over testing. The time 
it takes the individual to give the correct respdnse is measured, cad 
time-intervalB, e.g. zero to one second, one to two seconds, two to five 
seconds, five to ten seconds, more than ten seconds, correspond to responije 

31 



alternatives^ An individual who responds in three seconds has 'Va38ed•^ 
the ten-second and the five-second "alternatives" and failed the two- 
second and one-second ones. Thus the data is treated as if there were 
an ±tt|)lied logical relation ariiorig the alternatives. ^ 

These are some pdssibie methods of handjLing items which take ac- 
count of multipliB alternatives. Others may suggest themselves In the 
course of time and experience. One thing that Meds to be taken ^ac- 
count of in evaluating the consistency of items is that the domin^cer 
relations among alternatives to the same it «Jm are necessarily perfectly 
consistent, ihus they may inflate the oyer all consijStency estimates ► 
Thus V would have to be corrected for these intra-item dominance rela- 
tions, c ' ■' . f^.. . 
Incomplete Data 

That basic definitions of thfe number of actual i minimum/ and chance 
Item dominiEmce relations can be employed for Incomplete item data just 
t as they were for the total test. The problems and decisions 'are just 
the same. 

As noted before in the discussion of v/ v. is calculated from the 
njj^ rather than using the^ marginals. Similarly, v^^ is calculated ac- 
cording to its definition (86) rather than the simplification (87), once 
an order has been estaiiliihed. Finally; the E(n.- ) must be calculated 
on the basis of only those individuals who took both j and k rathe;^ than 
for all. These qualifications mean that the item indices are somewhiit 
more troublesome to compute in tire incomplete case, but this is unlikely 
to cause undue distress since the availability of the computer is assumed. 
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, The item statistics can be applied to the matrices that include im- 
plied relations as well a6 to the directly obtained ones. The caveat 
that the method of completing the relations matrices may glVe an ^rfclf 1- 
>&ial degree of apparent consistency should be borne In mind, however* 

* -Discussion 
Basic Rationale of the Approach > • 

The developments here followed primarily from cpnsidering the wrongs 
matrix as well as the rights matrix. At first glance, this is a trivial 
addition to test t^heory, but it has several advantages.- » 

The first advantage of considering items passing people as well as 
pefi^ple parsing Items is a rather fundamental pne. It provides a means of 
demonstrating that test data can produce an order in the formal sense, of 
a transitive asymmetric relation. If there Is a Gut tman scale, then 

Is upper triangular <;hen the elements- are suitably ordered* Con- 
sidered as the adjacency matrix , of the gjraph of a relation, It corresponds 
; to the adjacency matrix of an*order. Thtis the correspondence is vary 
readily anSP^learly iestalllished. , 

Tfie second conceptual advantage lies in the degree to which i£ empha- 
sizes the duality between person and item relations/ Surely we arc not 
interested in consistency of item ordering per se, but rather in conais- 
tency and completeness of person ordering. Item relations are a mGans ►to 
the end of person relations. Use of the complete item-person domisaance 
matrix facilitates the use of the duality in order to develop indices which 
reflect person relations. 

r 

Our primary mo^^^tion here has been toward the development t)f con- 
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aiatency indices which were applicable to tailored tests as well as to- 

Standard, ones • Obviously, in that case, the wrongs matrix Is not com- 
pletely redundaftt. There are three possible Item-per-son relations now; 
right/wrong, and not taken, rather than just two. What we have tried 
to show here Is that even in the incoiiiplete ^case there are item-item and 
person-person dominance relations and their consistency, and completeness 
can be evaluated using just the same £u»dantental quantities as in the 
complete case. 

Thus, a test theory based on the complete matrix of relations among ^ 
members^ of an extended set which includes both items and persons has a 
•number of conceptual Advantages. Not the least of these is that It pro-^ 
vldes measures of consistency for tailored tests which are based solely 
on the test Items as they are administered in the tailored context. 
StatlflticAl Considerations 

The developments here continue the unfortunate psychometric tradition 
of treating statistical data descriptlyely rather than inferentlally. 
Thete is no real justification for this except to say that an Inferential 
syB^em. is pointless unless what it is attempting to infer is felt to be 
worth knowing. Therefore, a set of quantities are -presented which gre 
argued to be important descriptors of a measurement system. Unleoo 'they 
are accepted as valid, the additional labor of dealing with the in:2arGn- 
tial problems that are involved is not justified. To atop at the descrip- 
tivc level If the descriptors are accepted is equally unjustified. 

All that will be done here in considering problems of inference is 
point out some of the statistical problems. First is the definition of 



the dimension of genetallzatton. An observed value of one of the con^- 
©latency Indices i,8^ba8ed oil a sample of It^ma and a sample of pesaons. 
In tailored situations. It Is furthermore based on a samplS of the. 11&- 
person relations In the sampleV Thus la addition to the problems of lii'- 
fetrlng the value of one of the consistency Indlce^ for the whole ^pu- 
latlon of Items from a sample of Items or for a whole population of per- 
sons from a sample of persons, 'we hjave that of Inferring from ^sample 
of person-item relations to a population of person-item relatidns. The 
fact that item-person relations are not sampled randomly in a tailored 
test but rather are chosen by s 096 scheme wjhlch hopes to optimize the 
selection makes the last inferential problem an especially thorny one» 
Evaluating the sampling characteristics 'by Monte Carlo method might ba 

nethod, 

tive to the design of the sampling experiment* 

There is at least one problem of probable bias that should he pointed 
out. This has to do with v^ and g^. Items and persons are placed in or- 
der on the basis of sample data, so the true order may be different from 
the sample one. The sample order minimizes v^ and g^, so some inflation 
can be expected in the population with consequent shrinkage in conoio- 
tency indices. A correction for this effect 'm^be feasible, at laast 
in the case of complete data, ^ 

It may well be that problems of statistical inference will not be 
soluble without the adoptiotl of explicit otatiotlcal model relatinc item- 
person relations to underlying parameters. This has been the route fol- 
lowed by modem test theorists who have followed the traceline or item 



the only feasible me^od, with conclusions that are consequently censi- 



characteristic curve routes (Blrnbaum, 1968; Lord and Novlck, 1968; Laz- 
arsf eld and Henry, 1968). This will be unfortunate since the accuracy 
of the inference will depei^d ;oh the accuracy of the statistical model, 
and £hat will be very difficult to assess • .] 
Relations between Item and Person Measures 

In view of the fact that the various v, g,,and c indices are based 

- ■ . * ■ 

on the same basic data; closer relations among them might be e9qpectcd> 

The only direct relation shown is that inter-relating v, g, v , and g , 

. c - . c 

Equation (94). ' , 

(94) ' ' 

Further work may well uncover additional ones. This would be a desirable 
outcome, particularly If it linked item and person indices more cloealy. 
That would permit the estimation; of one consistency index from the other, 
and might thus provide support for the use of item consistency to esti- 
mate person consistency. 

Review and Conclusions 
The fundamental premise here is that the set of binary relationa be-* 
tween persons and items which Is the outcome of the administration of a 
test Is usefully ^onsliered as a graph (cf» Harary, Cartwright, and Nor- 
man, 1963). The adjace^icy matrix of such a graph is items-plus-peroono 
by items-plus-persons with the rights matrix S as one part and the w^'ongs 
miatrlx/^' as a symmetrically plae<jd second part, as shown in Equation (1) 



(1) 

A *• 



0 W] 

B 0 



3G 



If the score matrix has Guttman forrn^ then a formal correspondence 



betw;een such a graph ^t:rlx and the graph matrix of a partial order, can 
be clearly demonatra^d. r • ' ' 

.The matrix contains the number of Item-Item and person-person 
dominance relations : .. - 



(4) N O 

a2 - % 

0 • X • 



0 



Then there are two fundamental measures of dominance 



(15) V - ZZn., ; 

and 



(16)' g - SSx • , 
Ih ^" 



where n^j^ wlll be the number of persons who get j wrong and k rdglitjr and 



x^^ will be the number of Items person 1 gets right which h gets wrong. * 

Jf the data are of Guttman form, then xt^^ «■ ^for J < k and Xj^^ « 0 

for 1 < h* In that case, v 'and g will take on the values v and g , re- 

m ®m' 

spectively, where , 

(21) / - E £ n , 

and . • 

(22) g - Six. 
• ^ 1 h>± 

If there Is, no dominance at all, then n^j^ « Uj^^ and ^ Xj^^: for all 
pairs of each* This suggests the following Indices of consistency o2 
Item (e^) and person (e ) relations, 

1/ * 37 ' • 



\- ■ .' ... . t • . ■ . V ■ ■■■ . ' ' 

(33) c -f5a , - " ► 

A tailored or computerized testing situation Is one where it can be 
true that s^j « 0 and » 0^ l.e, v the person dSes not taJke the item. 
A useful aspect of . the foJregoing measures and indices is that their def- 
inition is equally valid in that ease. 

Additional special cases of v arid g are. v and g and v and « . '^he 
and g^ are expected values of v and g under the assumption of indepen- 
dence: ' • . " " 

m . . 

I- • ■ . , 

(48) • ' V - £ S E(n ) , • , ■ ' : 

(64) 'g^ E ECk,.) 

1-h-l ^'V , ^ 

For complete data. 



and* 




(65) E(x,^) « 



i 



Ih*^ X 

where Uj is the number of people who get item j wrong and k right and 2c£ 
is the' number of items persm 1 gets dorrcct. In the/t*ilored caoo, the 
definitional of v^ and g^ stay\^ sawfe, but n^ , n and x must be re- 
defined so as to count only persons or .Items who are in common. \ 

Using v^ and g^> a pafr of consistency indices are 

* • -. ' ' , 

(73) . H^-IeJH^ , 
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. ■ • . , ■ -35- 

. " (74* Hp " g-g^ 

Is identical to Loevinger's (1947) homogeneity index. Again, they . 
are defined in both coaplete eind incomplete cases. 

The qulantities and jg^ are the paxlmum value§ of v and g. These 
are clearly defined only in the complete case since for incomplete data 

the definition of a maximum would require elaborate analysis. For com- 

V ■ ■ ' . '.• ' • ■ 

plete data, they are 

"I ■ ^ ' ' ' - - . ■ , 

(45) v„ - nJe(x - 5) • 

, — • •' ' ''\ • , 

and g^ is defined similarly. . . 

The f^lliar KR20 formula con be expressed'' In these terms 



(53) KR20 - X ""c 



1 V >^ V 
w 



This suggests a person consistency ifidex parallel to KR20, 

(62) KR 20 n " ^ 

• ' ■ P n - 1 



Thus the KR formulas eimnot be generalized to the tailored q^ae unless a/ 
means can be found to generalisse v • 



< 



Eor cbmplfete data, there are other simple and important relatione 
(42) V - £x^(x - x^) - ^ 

and 

(4i) g - 2nj(n - n^) . ' ^ 



a" 
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Also, 



(25) 



and 



(27) 



g - Z 

1 



m 7(x - 2± + l)x. 



CDtnplete^ness of ordering'^ia,,.^^ as important as conaistency* Thm 
two completeness Ittdidlfes were suggested 



(38) rp « Ztxl^ 



^in(n I) 



(39) 



J2EX(X - 1) 



They are usable^ In comj^lete as well as ineomplete eases • 

Item indices of consistency were also proposed. Using only thfoc 

parts of the sums v, v^, and y which refer to item J> indices parallel 

> m c 

• « 

to e^ and were derived 

• * 

(83) - 2vm^ 

v: tg 



and 



(90) 



mj cj 



These can be oscij for , the selection, deletion, and evaluation of itemn 

s 

In either the complete or ineomplete ease* . 

It was also pointed out that the mqasures could bo applied to data 
where, In the tailored case, some of the olcmcnto of the score matrix 
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have been entered by implication- rather than directly. This proeeduJ''e has 
been suggested by Cliff (1975) as an approach to computer-interactive, 
testing. It was hoted, however, that the methods for filling in entries 
In one of the score matrices may induce an artificial degree of cona?s-' 
tency to tjie data. . 

. Several aspects ^ of the utility of the system proposed here wer'fe 
dlacussed along with suggested methods of handli^pg certain kinds of data-. 
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